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PROGRESSIVE ADAPTIVE TIME STAMP 
RESOLUTION IN MULTIMEDIA AUTHORING 

CROSS-REFERENCE TO RELATED APPLICATION 

This application is continuation-in-part of provisional patent 
application Serial No. 60/106,764 filed November 3, 1998, the benefit of the 
filing date of which is hereby claimed for the commonly disclosed subject 
matter. 

DESCRIPTION 
BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention generally relates to composing and playing 
multimedia presentations and, more particularly, to a flexible time stamp 
information carried in the stream descriptor of the multimedia presentation. 

Background Description 

Multimedia authoring systems exist that allow the user (i.e., the 
author) to insert multimedia objects, such as video, audio, still pictures, and 
graphics, into a multimedia presentation at a certain spatial position and with a 
certain temporal location. Such an authoring system is used typically to create 
presentations that are in an MPEG-4 (Motion Picture Experts Group, version 



4) or SMIL (Synchronized Multimedia Integration Language) format. 

In more advanced authoring systems, the temporal location of the 
multimedia objects need not be absolute in time, but can be defined relative to 
other multimedia objects. This means that, for example, a video clip can be 
authored to start at the same time that a specific audio clip starts. Another such 
example is that after completely playing a certain video clip, another video 
clip should be played, possibly with some delay. The essence of this is that 
multimedia objects have start and end times that are defined with respect to 
the start and end times of other multimedia objects, with possible temporal 
offsets (delays). 

A further feature of advanced temporal authoring of multimedia 
objects is the possibility to have a range in duration of multimedia objects. For 
example, a certain video clip has a certain duration when played at the speed 
at which it was captured, say thirty frames per second. This now allows 
authors to define a range in the playback speed, for example between fifteen 
frames per second (slow motion by a factor of two) and sixty frames per 
second (fast play by a factor of two). This results in respectively a maximum 
and minimum total playback duration. In general, the advanced authoring 
systems allow authors to specify such ranges in multimedia object playback 
duration. Note, that it is still possible to dictate only one specific playback 
duration (which is directly related to the playback speed in the case of video, 
audio, or animation) by restricting the duration range to a zero width. 

If we now combine the relative start and end times of multimedia 
objects in the authoring system with the possibility to also specify a duration 
range, we see that a complete authored multimedia presentation is a complex 
but flexible system of interconnected objects with variable durations. The 
advantage of having this flexibility in duration lies in the data transmission 



and playback of multimedia objects. By not having very strict multimedia start 
and end times, the system has some flexibility to adapt to data delivery 
problems, which may be due to network congestion or transmission errors. For 
the final delivery and playback the system (which may be the server or the 
client) will resolve the true multimedia object start and end times during 
transmission and playback adaptive to the environment. 

In general, with these variable object durations, many actual values for 
start and end time are possible for all of the multimedia objects, especially 
when no delivery problems occur. In actual playback, absolute time stamps 
must be used. That means that for every multimedia object a playback 
duration is chosen which lies within the range of its possible durations. The 
problem of determining these factual durations at run time (i.e., playback) is 
addressed here. The method will be progressive in time; that is, it resolves the 
absolute time stamps as time advances, making it adaptive to the changing 
environment. Finally, it must be defined what information is to be sent to a 
client, that is sufficient to do the time stamp resolution. 

SUMMARY OF THE INVENTION 

It is therefore an object of the present invention to provide a technique 
for determining the factual durations of multimedia objects at run time 

It is another object of the invention to provide a new dedicated 
descriptor of object time duration to alleviate the problem of unreliable 
delivery of objects in a multimedia presentation. 

According to the invention, the solution to the problem consists of two 
parts. First, it is necessary to define what information must be available to the 
client in order to be able to determine the multimedia object durations. And 



second, the resolution of the durations themselves must be solved. The new 
flexible timing information can be used by the client to adapt the timing of the 
ongoing presentation to the environment, while having more room to stay 
within the presentation author's intent and expectations. 

Six steps are used to resolve the actual label time, and the 
corresponding duration of the multimedia objects that have that label for their 
respective end times. In the first step, all the dependency relations are 
collected for the label Px 5 by taking all objects n that have Px as the label for 
their end time: 

t n + minimum(w) z t x <> t n + maximizing) n = 1, . . ., N , 
Here t n is the start time of object w, and JVis the number of objects. 

In the second step, the iV relations are used to calculate the tightest 
bounds on t x : 

min{^} ^ t x z max{*J 

with 

min{^} = max {t n + minimum(w)} n = 1, . . N 
max{£j = minfo + maximum(w)} n = 1, . . ., N 
In the third step, the bounds on the durations of each object n are 
recalculated by using: 

duration(/i) — t x -t n 

to get 

min{^} - t n <> duration(n) <, max{^} - t n n = 1, . . ., N 
In the fourth step, the preferred duration of each object n is 
recalculated: 

if (preferred(w) < min{t x } - t n ) then 

preferred(«) = mm{t x } - t n 
else if (preferred(«) > max {/J - Q then 



f 



5 



preferred(w) =max{^} - t n 

end if 

In the sixth step, the general error criterion for resolving the duration 
of each multimedia object is defined as: 



N 

£ {duration(n) -preferred(w)} 2 



or, substituting duration(«) = t x - t a : 



E= £ {^-^-P ref erred(/2)} 2 



If we take the derivative of E with respect to t x , and set this to 0, we see that 
the optimal solution for the absolute time t x of label Px is: 

1 N 

77E {^ + preferred(«)} 



Finally, in the sixth step, the corresponding duration of multimedia 
object n is calculated with: 
duration(w) = 



BRIEF DESCRIPTION OF THE DRAWINGS 



The foregoing and other objects, aspects and advantages will be better 
understood from the following detailed description of a preferred embodiment 
of the invention with reference to the drawings, in which: 

Figure 1 is a block diagram of one preferred computer system with 
multimedia inputs and outputs that uses the method of the present invention; 



Figure 2 is a temporal diagram illustrating the problem solved by the 
present invention; 

Figure 3 is a flow diagram showing the logic of the overall process 
according to the invention; 

Figure 4 is a flow diagram showing the logic of the process for 
calculating the minimum and maximum times in block 302 of Figure 3; 

Figure 5 is a flow diagram showing the logic of the process for 
calculating t x in block 303 in Figure 3; and 

Figure 6 is a flow diagram showing the logic of the process for 
calculating the durations of the objects in block 304 of Figure 3. 



DETAILED DESCRIPTION OF A PREFERRED 
EMBODIMENT OF THE INVENTION 



Referring now to the drawings, and more particularly to Figure 1, there 
is shown in block diagram form a computer system 1 00 on which the subject 
invention may be practiced. The computer system 100 includes a personal 
computer (PC) 105 running a windowing operating system and including a 
multimedia audio/video capture adaptor 1 10. A video camera 122 connects to 
the adaptor 1 10 as does an optional playback monitor 124 for multimedia 
presentations composed on the computer system 100. Other multimedia 
hardware 130 may be included as well as various input devices, such a 
keyboard (not shown), a cursor pointing device (e.g., a mouse) (not shown) 
and a microphone 132 or other audio input device, and a monitor 134 on 
which a graphic user interface (GUI) of the operating system and application 
software is displayed. The computer 105 includes secondary memory storage 
(e.g., a hard drive) 140 of adequate capacity to store the multimedia 
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presentation being authored. 

The solution to the problem outlined above is best illustrated by a 
simple example. Let us consider a presentation that is authored having three 
multimedia objects, a video clip (V), an audio clip (A), and a background 
image (B). As explained above, the Isis authoring system requires the author 
to specify for each multimedia object the duration range, as well as a relative 
start and end time. For the three objects in our exemplary presentation, the 
parameters are authored as: 





start 


end 


minimum 
duration 


preferred 
duration 


maximum 
duration 


V 


PI 


P2 


3 seconds 


4 seconds 


5 seconds 


A 


P2 


P3 


3 seconds 


4 seconds 


4 seconds 


B 


PI 


P3 


7 seconds 


7 seconds 


8 seconds 



The labels PI, P2, and P3 are to indicate how the various multimedia objects 
are temporarily related. This means, for example, that objects V and B start at 
the same time. The temporal aspect of this authored presentation can be 
depicted more clearly in Figure 2. 

As shown in Figure 2, the background image B starts a point PI and 
ends at a point P3. The duration times are shown in brackets as 7,7,8 
corresponding to 7 seconds minimum duration, 7 seconds preferred duration, 
and 8 seconds maximum duration. Similarly, the video clip V begins at the 
point PI and ends at a point P2, and the audio clip A begins at the point P2 
and ends at the point P3, again with duration times shown in the brackets. 

The player (the client) of the multimedia presentation first receives the 
multimedia object parameters for video clip V and background B. The player 
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then initializes the time of point PI (arbitrarily) to ^=0, and starts playing the 
two objects V and B with their preferred duration. For the video clip V, this 
means it will be played at the corresponding preferred speed. If no network or 
playback delays occurred, the video will finish after four seconds. However, if 
5 a delay of Vi second occurred during playback, the time of point P2 is not t 2 z =4 9 
but t 2 =4.5. The player next attempts to resolve the durations of B and A. It 
does this using the relations: 

t x + 7 < t 3 <, t x + 8 
t 2 + 3 < f 3 £ f 2 + 4 

10 Knowing that f,=0 and t 2 -4.5 9 we obtain: 

7 <; t 3 <; 8 
7.5 <; t 3 <; 8.5 

Which is combined into: 

7.5 < t 3 < 8 

15 With this we can recalculate the duration range for both the background B and 

audio clip A. Using: 

duration(B) = t 3 - t x — t 3 
duration(A) = t 3 - 1 2 = t 3 - 4. 5 

we get 
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7.5 <; duration(B) <; 8.0 
3.0 £ duration(A) < 3.5 

We next use these new duration ranges to redefine the preferred durations of 
both audio clip A and background B. For background B, we see that the 
preferred duration cannot be met, and we have to settle for the closest value to 
the preferred value, which is now 7.5 seconds. Similarly, the preferred 
duration for the object audio clip A changes to 3.5 seconds: 

preferred(B) = 7.5 
preferred(A) = 3.5 

Finally, we can use these now feasible preferred durations to determine a good 
value for the time * 3 at point P3, and thus for the durations of the objects B and 
A. We do this by defining an error criterion on the durations as the sum of the 
squared deviations from the (updated) preferred durations: 

E = {duration(B) - preferred(B)} 2 + {duration(A) - preferred(A)} 2 

Using the definitions of the durations from above, and the recalculated 
preferred durations, this is rewritten into: 

E= {h-1.5} 2 + #3-4.5-3. 5} 2 = fe-7.5} 2 + {h-%$} 2 

Minimizing this error with respect to f 3 simply yields: 

* 3 = %(7.5+8.0) = 7.75 
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and the durations are 

duration(B) = 7.75 
duration(A) = 3.25 

From this example, it will be understood that the solution to the 
5 problem consists of two parts. First, it is defined what information must be 
available to the client in order to be able to determine the multimedia object 
durations. And second, the resolution of the durations themselves must be 
solved. 

A client (i.e., player of the multimedia presentation) must receive for 

10 each multimedia object five items of information. These items are the two 
labels, one for the object's start time and one for the end time, and the three 
durations, the minimum, maximum, and the preferred duration. In the case of 
video, audio, and other multimedia objects that have a playback speed, the 
preferred duration must correspond to the "regular" playback speed of the 

1 5 object. The information on a particular multimedia object must be delivered to 
the client prior to starting playback of the object. 

When playback has finished for a particular multimedia object, the 
absolute time of a certain label will become known. This means, that one or 
more label times can be resolved using this new information. The time stamp 

20 resolution is therefore progressive over time, as more information becomes 
available in the form of factual multimedia object durations, and arrival of 
information of objects that are to be played in the (near) future. 

To resolve the actual label time, and the corresponding duration of the 
multimedia objects that have that label for their respective end times, the 

25 following steps are taken: 
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Collect all the dependency relations for the label Px, by taking all 
objects n that have Px as the label for their end time: 

t n + minimum(«) zt x zt a + maximum^) n = 1, . . ., N 
Here t n is the start time of object n, and Wis the number of objects. 
Use the N relations to calculate the tightest bounds on t x : 

min{/J < t x <> max {4} 

with 

minfo} = max {4 + minimum(n)} n = 1, . . N 
maxfo} = min{/„ + maximum(«)} n = 1, . . N 
Recalculate the bounds on the durations of each object n, by using: 
duration^*) = t x -t n 

to get 

minfc} - /„ <> duration(n) ^ max {/J -t m n = 1 ? . . N 
Recalculate the preferred duration of each object n: 
if (preferred(n) < mm{t x } - Q then 

preferred(n) = minfo} - t n 
else if (preferred(«) > max{^} - /„) then 

preferred(«) =max{YJ - t n 

end if 

The general error criterion for resolving the duration of each 
multimedia object is defined as: 

N 

E= ^ {duration(n) - preferred(n)} 2 

or, substituting duration(«) = t x -t n \ 

E= J) {4-/ n -preferred(«)} 2 
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If we take the derivative of E with respect to t x , and set this to 0, we 
see that the optimal solution for the absolute time t x of label Px is: 

1 N 

t x = — £ {f„ + preferred(n)} 

6. The corresponding duration of multimedia object n is calculated with: 
duration(w) = t x -t n 

The entire process of steps 1 through 6 is summarized as illustrated in 
Figure 3. The inputs to the process as in step 1, supra, are shown at block 301. 
Step 2 calculates the minimum and maximum end times over all multimedia 
objects in function block 302. This is described in more detail in the 
description of Figure 4, infra. Next, the steps 3, 4 and 5 are combined in 
function block 303. This is described in more detail in the description of 
Figure 5, infra. Finally, the durations of the objects are calculated in function 
block 304, which is described in more detail in the description of Figure 6, 
infra. 

Step 2 (i.e., block 302 of Figure 3) is illustrated more detail in Figure 
4. The process is initialized in function block 401 before entering the 
processing loop. The value of n is incremented by one in function block 402 at 
the beginning of the processing loop. A test is made in decision block 403 to 
determine if the minimum end time is less than the start time of object n plus 
the minimum duration of object n. If so, the minimum time is set to that value 
in function block 404. If not, a test is made in decision block 405 to determine 
if the maximum end time is greater than the start time of object n plus its 
maximum duration. If so, the maximum time is set to that value in function 
block 406. Finally, a test is made in decision block 407 to determine if all 
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objects have been processed and, if not, the process loops back to function 
block 402 where the value of n is again incremented, and the maximum and 
minimum times for the next multimedia object are calculated. This processing 
continues until the minimum and maximum end times over all N multimedia 
objects have been calculated. 

Steps 3, 4 and 5 (i.e., block 303 in Figure 3) are illustrated in more 
detail in Figure 5. The process is initialized in function block 501 before 
entering the processing loop. The value of n is incremented by one in function 
block 502 at the beginning of the processing loop. A test is made in decision 
block 503 to determine if the preferred duration is greater than the minimum 
end time less the start time of a current object n. If not, the preferred duration 
is set to this value in function block 504; otherwise, a further test is made in 
decision block 505 to determine if the preferred duration is less than the 
maximum end time less the start time of the current object w. If not, the 
preferred duration is set to this value in function block 506; otherwise, the 
preferred duration is set to the preferred duration of the object n in function 
block 507. Then, in function block 508, the sum of the times is calculated. A 
test is made in decision block 509 to determine if all objects have been 
processed and, if not, the process loops back to function block 502 where the 
value of n is again incremented. When all objects have been processed, the 
time t x is computed as the sum divided by N, the number of the multimedia 
objects, in function block 510. 

Step 6 (i.e., block 304 in Figure 3) is shown in more detail in Figure 6. 
The process begins by initializing n to zero in function block 601. The value 
of n is incremented by one in function block 602 at the beginning of the 
processing loop. The duration of each object n is calculated in function block 
603 as the calculated time t x minus the start time t{n) of the object n. After 
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each calculation, a test is made in decision block 604 to determine if all 
objects have been processed. If not, the process loops back to function block 
602 where n is again incremented and the duration of the next object is 
calculated. The process ends when all N objects have been processed. 

While the invention has been described in terms of a single preferred 
embodiment, those skilled in the art will recognize that the invention can be 
practiced with modification within the spirit and scope of the appended 
claims. 



f 
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CLAIMS 

Having thus described our invention, what we claim as new and desire 
to secure by Letters Patent is as follows: 



1 1 . A method of progressive time stamp resolution in a multimedia presentation 

2 comprising the steps of: 

3 supplying a player of a multimedia presentation with information 

4 comprising two labels, one for a multimedia object's start time and one for the 

5 multimedia object's end time relative to other multimedia object start and stop 

6 times, and three durations, a minimum duration, a maximum duration and a 

7 preferred duration for each multimedia obj ect prior to starting playback of the 

8 multimedia object; and 

9 resolving the durations of multimedia objects using said information 

10 based on actual multimedia object durations and arrival of information of 

1 1 multimedia objects to be played. 

1 2. The method of progressive time stamp resolution in a multimedia 

2 presentation recited in claim 1 wherein the step of resolving comprises the 

3 steps of: 

4 calculating minimum and maximum end times for over all multimedia 

5 objects; 

6 calculating actual end times that are shared by all multimedia objects; 

7 and 

8 recalculating a preferred duration of each multimedia obj ect. 
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3. The method of progressive time stamp resolution in a multimedia 
presentation recited in claim 1 wherein the step of resolving comprises the 
steps of: 

collecting all the dependency relations for the label Px, by taking all 
objects n that have Px as the label for their end time: 

t n + minimum(w) zt x £t n + maximum^) n = l 9 ... 9 N 
where t„ is the start time of object n, and N is the number of objects; 
using the N relations to calculate the tightest bounds on t x : 
minfo} ^ t x <, maxfo} 

with 

min{/J = max{/„ + minimum(w)} n = 1 , . . N 
maxfe} = minfo + maximum(n)} n = 1, . . N; 
recalculating the bounds on the durations of each object «, by using: 
duration(n) = t x - t n 

to get 

minfo} - t n <> duration^) <, maxfo} -t n n = 1, . . N; and 
recalculating the preferred duration of each object n according to the 
process: 

if (preferred^) < minfo} - Q then 

preferred(w) = minfo} - t n 
else if (preferred(w) > max{fj - t n ) then 

preferred(«) =max{t x } - t n 

end if. 

4. The method of progressive time stamp resolution in a multimedia 
presentation recited in claim 3 wherein the step of resolving further comprises 
the steps of: 
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using as the general error criterion for resolving the duration of each 
multimedia object: 

N 

E= 53 {duration(w) - preferred(«)} 2 

or, substituting duration(«) = t x - t n : 

E= £ fo-^-preferred(w)} 2 

and taking the derivative of E with respect to t„ and setting this to 0 to obtain 
the optimal solution for the absolute time t x of label Px as: 

1 N 

t x = — £ {^ + prefeired(w)};and 



calculating the corresponding duration of multimedia object n as: 
duration^) = t x - t n . 
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PROGRESSIVE ADAPTIVE TIME STAMP 
RESOLUTION IN MULTIMEDIA AUTHORING 

ABSTRACT OF THE DISCLOSURE 

Environments with unreliable delivery may result in faltering 
5 presentation of multimedia objects, due to missing time stamp deadlines. This 
may be alleviated by introducing more flexible time stamping. To avoid this, 
additional MPEG-4 object time information is sent to the client. This requires 
a new dedicated descriptor, carried in the Elementary Stream Descriptor. The 
new more flexible timing information will have two features. First, instead of 
10 fixed start and end times, the duration of an object can be given a range. And 
second, the start and end times are made relative to other multimedia object 
start and end times. This information can then be used by the client to adapt 
the timing of the ongoing presentation to the environment, while having more 
room to stay within the presentation author's intent and expectations. 
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I acknowledge the duty 10 disclose information which is material to the examination of this application In accordance 
with Title 37, Code of Federal Regulations, § 1.56(a).* 

I hereby claim foreign priority benefits under Title 35, United States Code, $119 of any foieign application^) for 
parent or inventor's certificate listed below and have also identified below any foreign application for patent or investor's 
certificate having a filing date before that of the application on which priority is claimed: 

Prior Foreign Applicarioax(s) Priority Canned 



(Number) (Country) (Day/Mondt/Year Filed) yes 



(Number) (Country) (Day/Month/Year Filed) yes no 

I hereby claim the benefit under Title 35, United States Code, § 120 of any United States application© listed below 
and, insofar as the subjecr matter of each of the claims of this application is not disclosed in the prior Uniied States ^plication 
in the manner provided by the first paragraph of Tide 35 T United Stares Code, § 112, 1 acknowledge the duty to disclose 
material information as defined in Tide 37, Code of Federal Regulations, §1 .36(a) which occurred between the filing date of 
the prior application and the national or PCT international filing date of this application: 

MWXW ..JUflP/gg- Pending Pmvku^ 

(Application Serial No.) (Piling Dare) (Scam*: patented, pending, abandoned) 

Power of Attorney: As a named investor, I hereby appoint Manny W. Schectcr, Keg. No. 31,722, Tory J. Uardi, 
Reg* No. 29,936, Stephen C. Kaufman, Reg- No. 29,551, Louis J, Pexcdlo, Reg, No. 33.206, Jsy p. Sbtollini, Reg. No. 
36,266. Robert M. liepp, Reg- No, 25,933, Daniel P. Montis, Reg. No. 32,053, Kevm P. Jordan, Reg. No. 40,277 % Douglas 
W. Cameron, Reg. No. 31,596, David M. Shofi, Reg, No. 39>835, Christopher A. Hughes, Reg. No. 26 ( 9l4 t Edward A. 
Pennington, Reg, No. 32,5SS, John E. Hod, No, 26,279, Joseph C. Redmond, Jr., Reg, No 18,753, C. Lament 
Whitham, Reg. No. 22.424, Marshall M Cunts, Reg. No. 33,138, and Michael E. Whitham, Reg. No. 32,635, as attorneys 
and/or agents to prosecute this application and transact all business in the Parent and Trademark Office connected therewith. 
All correspondence should be directed to Whitham, Curtis & Wfcxtfcam, Resion International Center, 11800 Sunrise Valley 
Drive* Suite 900, Reston, Virginia 20191* Phone calls should be directed to Whitham, Curtis AWhirfuan, ai7Q3/39l-25io. 



NOV 25 '98' 11: 31 FR NETWORKED COMPUTING 914 784 6485 TO 917033919035 
" w rivurrrrriinftiii iuiu la&irm JMAM 70339I9J35 



P. 03/05 
T-358 P.Zt/tt F-531 



Docket No.: YC&48-446 



[hereby declare that all statements made herein of my own knowledge axe true and thai all statements inadeoa infonoarion and 
belief a*e believed to be true; and further that ttese statements were m*k with the knowledge that willful false statements and 
the like so made are punishable by fine or imprisonment, or both, under Section 1001 of Tide 18 of xbe United States Code and 
thai such willful false statements may jeopardize the validity of the application or any patent issued thereon. 




(1) Inventor 
Signature: 

Residence: 23 Pheasant Run, Scarsdale, New York 10583 
Citizenship: United States of America 
Post Office Address: Same as above 

(2) Inventor: Peter H,/ 
Signature; 

Residence: 3$ \& WoJdea Road, Apt. C-l-3, Ossming, New York 10562 
Citizenship: Netherlands 
Post Office Address: Same as above 



Date; 



Mae „ . 

Notary Public, State of New York 
No. 01LE6003456 
County of Westchester 
Commission Expires 02 2000 




Dare: 



MaeELLemke 
Notary Public, State of New York 
No. 01LE6003456 
County of Westchester 
Commission Expires March 02 2000 



*Tid<?37, Code of Federal Regulations, §L56(a): 

(a) A duty of candor and good faith towazd the Patent and Trademark Office rests on the inventor on each attorney or agent 
who prepares or prosecutes the application and on every other individual who i$ substantively involved in the preparation or 
prosecution of the application and who is associated with ihe inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. All such individuals have a duty to disclose to the Office infonnaiion they are aware of 
which is material to the examination of the application. Sucft information is material where there is substantial likelihood thai a 
reasonable egatniner would consider it important in deciding whether to alio w the application to issue as a patent . The duty is 
commensurate with die degree of involvement in the preparation or prosecution of the application. 



(b> Under this section, information is material to patentability when it is not cumulative to information already of record or 
being made of record in the application* and (1) it establishes, by itself or in combination with other information, a prima facie 
case of unpatentability; or (2) it refutes, or is inconsistent with, a position the applicant takes in: (i) opposing an argumenj of 
unpatentability relied on by the Office, or (ii) asserting an argument of patentability. 



- % 



vim i * rw-k * 



